A Pascal Program for the Kalman Learning Algorithm in Multilayer Neural Networks

In recent years, neural networks have been applied widely in industry. The most commonly used model is the backpropagation (BP) multilayer neural network (Ryan, Sementilli, and Hunt, 1991; Karunanithi, Whitey, and Malaiya, 1992; Ohga and Seki, 1993; Fan, Nikolaou, and White, 1993; Huang and Lindqvist, 1995; and Malmgren and Nordlund, 1996). Among the drawbacks of the BP neural network are local minima and slow convergence as a result of lack of optimality in selecting the learning parameters associated with the different weights and the complex, nonconvex error function in the weight space (Cho and Don, 1991). Many attempts have been made to solve these problems. One proposed solution was to introduce the Kalman filter theory (Kalman, 1960) as a learning law in multilayer neural networks by Singhal and Wu (1989). They considered that the training of multilayer neural networks is an identification problem for a nonlinear dynamic system, which can be solved using the extended Kalman theory. Shah and Palmieri (1990) and Shah, Palmeiri, and Datum (1992) discussed the multiple extended Kalman algorithm through multilinear parameterization to localize the computation of Singhal and Wu’s algorithm. The corresponding recursive estimate of the optimal weight vectors is given by Haykin (1994). Cho and Don (1991) developed a Kalman algorithm for fast learning in muitilayer neural networks which updates the weight vector in each neuron. This new updated weight vector is consistent with accumulated information and can lead to a minimizion of the trace of the training error covariance matrix for each neuron. Andrej and Andrej (1995) proposed a generalised BP algorithm in which the synaptic covariance matrix, which is derived from Kalman filter theory, is introduced to each neuron in the network through a learning rate adaptation. All these studies show that the introduction of the Kalman theory to the training process of multilayer neural networks results in a lower chance for the network to become trapped in local minima than the BP algorithm. In this paper, a Kalman learning algorithm, which combines all previous studies of the Kalman theory’s learning law in multilayer neural network with learning rate adaptation, is proposed and the computer source code is presented. The key idea in the Kalman learning algorithm for adjusting weights in the iteration process is the gain matrix. This gain matrix is an optimal convergence factor and a more global way of learning in multilayer neural networks because all the information available from the start of the training process up to the current training sample is exploited in the update procedure for weight vectors, The Kalman learning algorithm is different from the BP algorithm, which only calculates the differential of the error to the corresponding weight. The Kalman learning algorithm multilayer neural network was applied to estimate ore grades from geophysical logging data in a Swedish mine. The results obtained from the Kalman learning algorithm are compared with those obtained from a BP algorithm network.